Distributed systems allow multiple clients in a network to access a pool of shared resources. For example, a distributed storage system allows a cluster of host computers to aggregate local disks (e.g., SSD, PCI-based flash storage, SATA, or SAS magnetic disks) located in or attached to each host computer to create a single and shared pool of storage. This pool of storage (sometimes referred to herein as a “datastore” or “store”) is accessible by all host computers in the cluster and may be presented as a single namespace of storage entities (such as a hierarchical file system namespace in the case of files, a flat namespace of unique identifiers in the case of objects, etc.). Storage clients in turn, such as virtual machines spawned on the host computers may use the datastore, for example, to store virtual disks that are accessed by the virtual machines during their operation. Because the shared local disks that make up the datastore may have different performance characteristics (e.g., capacity, input/output per second or IOPS capabilities, etc.), usage of such shared local disks to store virtual disks or portions thereof may be distributed among the virtual machines based on the needs of each given virtual machine.
This approach provides enterprises with cost-effective performance. For instance, distributed storage using pooled local disks is inexpensive, highly scalable, and relatively simple to manage. Because such distributed storage can use commodity disks in the cluster, enterprises do not need to invest in additional storage infrastructure. However, one challenge that arises relates to developing a mechanism to efficiently track where objects are stored across the commodity disks in the cluster as well as how to efficiently access them when needed. For example, while utilizing a flat namespace may provide a simplistic and efficient means to store and retrieve objects, it does not provide enough flexibility to create hierarchical relationships between objects that may be useful in organizing objects in a manner that is compatible with the existing interfaces of clients or that otherwise satisfies the different storage requirements of different clients. For example, many pre-existing environments that could leverage such a scalable object store (e.g., applications, storage management tools, virtualization hypervisors, etc.) may require that the object store provide a certain hierarchical file system based storage interface. One example is vSphere Hypervisor from VMware, Inc. which stores virtual machine metadata in the form of files in a hierarchical file system. Furthermore, because the datastore is shared among the cluster of host computers which may simultaneously access the same data stored therein, any file system that is used to manage the single namespace provided by the datastore needs to have mechanisms for concurrency control. Current distributed or clustered file systems typically provide some form of concurrency control. However, due to limitations in their inherent design, such current file systems typically have limitations on the number of simultaneous “clients” (e.g., host computers or virtual machines that access the file system) they can support. If a current file system used to manage the datastore has limits on the number of clients that can simultaneously access it, then the utility of a highly scalable datastore can plateau, even if additional commodity storage can be easily added to the datastore to increase its capacity, since no additional clients can be added to access such additional storage.